{
 "cells": [
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "#pandas有两个主要的数据结构：Series和DataFrame。Series是一维数组，可以包含任何数据类型，DataFrame是二维表格，包含多个Series。",
   "id": "87f6f51881baf11e"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ],
   "id": "initial_id",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "#Series:由一组数据（ndarray一维数组）以及与之对应的索引组成。索引自动创建，不指定索引时，默认从0开始。",
   "id": "799db7a6ccfc8356"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "##创建Series对象pd.Series()",
   "id": "fd369229a6117ee2"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "my_series = pd.Series([6,7,8,9,10,11,12,13])#通过列表创建建Series\n",
    "print(type(my_series))\n",
    "print(my_series)#打印输出Series会带有其数据类型my_series.dtype\n",
    "my_series=pd.Series(range(6,14))#通过range函数创建Series\n",
    "my_dict={'a':1,'b':2,'c':3,'d':4}\n",
    "new_series=pd.Series(my_dict)#通过字典创建Series。字典的键成为索引，值成为数据。\n",
    "print(new_series)"
   ],
   "id": "371b1d0dfab63e97",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "##索引与数据的对应关系不被运算影响",
   "id": "cdef51690724661d"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "print(my_series)\n",
    "print(my_series*2)\n",
    "print(my_series>10)#关系运算后返回的Series中数据部分只有True和False（数据类型为bool）\n",
    "print(my_series[my_series>10])#返回>10的一个Series对象，索引不变。\n",
    "print(my_series.map(lambda x : x**2))#根据map传入的函数对数据进行运算。"
   ],
   "id": "bec3ae6f7b99250f",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "##异常值的替换",
   "id": "44297738de9da54"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "print(my_series)\n",
    "print(my_series.replace(6,100))#6替换为100\n",
    "print(my_series.replace([6,8],100))#6和8替换为100\n",
    "print(my_series.replace([6,8],[100,200]))#6替换为100，8替换为200"
   ],
   "id": "206dca162c078d0a",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "##通过索引获取数据",
   "id": "2ca054372cc67007"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "print(new_series)\n",
    "print(new_series.iloc[0])#使用索引位置取数据（用iloc方法）\n",
    "print(new_series.loc['a'])#使用索引标签取数据（用loc方法）\n",
    "print(new_series.iloc[1:4])#使用索引位置切片（左闭右开）\n",
    "print(new_series.loc['b':'d'])#使用索引标签切片（左闭右闭）"
   ],
   "id": "8390aafb97c78027",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "",
   "id": "c6acb0b0b55cf75e"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "#DataFrame:表格型数据结构，既有行索引(index)又有列索引(columns)。索引自动创建，不指定索引时，默认从0开始。",
   "id": "90f0a4354871f90d"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "##创建DataFrame对象pd.DataFrame()",
   "id": "a8c48f0789db8549"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "my_dataframe = pd.DataFrame([[0,1,2,3],[4,5,6,7],[8,9,10,11]])#通过二维列表创建DataFrame\n",
    "print(my_dataframe)\n",
    "my_dataframe=pd.DataFrame(np.arange(12).reshape(3,4))#通过二维数组创建DataFrame\n",
    "print(my_dataframe)\n",
    "my_dict=[{\"name\" : \"xiaohong\" ,\"age\" :32,\"tel\" :10010},{ \"name\": \"xiaogang\" ,\"tel\": 10000} ,{\"name\":\"xiaowang\" ,\"age\":22}]#列表内套字典创建DataFrame，可以有缺失值，缺失值在创建时自动用NaN填充。\n",
    "# my_dict={\"name\":[\"xiaohong\",\"xiaogang\",\"xiaowang\"],\"age\":np.arange(20,23),\"tel\":[10010,1000,10011]}#这种在创建时不允许有缺失值\n",
    "my_dataframe=pd.DataFrame(my_dict)#字典的键成为列索引，值成为数据。\n",
    "print(my_dataframe)#DataFrame中同一列必须是相同数据类型，不同列可以是不同的数据类型。\n",
    "print(my_dataframe.dtypes)#查看每一列的数据类型"
   ],
   "id": "a005e89ccb892ff5",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "##日期数据类型datetimes",
   "id": "bcc3f621a9469acd"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "dates = pd.date_range(\"20250101\", periods=6, freq=\"D\")  #通过pd.date_range函数生成日期序列。freq默认为'D'（天）\n",
    "new_dataframe = pd.DataFrame(np.arange(24).reshape(6, 4), index=dates, columns=list(\"ABCD\"))  #通过index指定行索引columns指定列索引\n",
    "print(new_dataframe)\n",
    "print(new_dataframe.head(2))  #通过head方法指定获取前几行，默认获取前5行"
   ],
   "id": "5d28c17721de8ee1",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "##获取索引",
   "id": "e3b1690fa647f81f"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "print(my_dataframe.index)#获取行索引\n",
    "print(my_dataframe.columns)#获取列索引"
   ],
   "id": "4425c9724c6356f4",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "#增加列数\n",
    "my_dataframe[\"A\"]=np.arange(3)#自定义列索引，并赋值\n",
    "print(my_dataframe)\n",
    "#删除列\n",
    "del my_dataframe[\"A\"]\n",
    "print(my_dataframe)\n",
    "#改列名\n",
    "my_dataframe=my_dataframe.rename(columns={\"tel\":\"phone\"})\n",
    "print(my_dataframe)\n",
    "my_dataframe=my_dataframe.rename(columns={\"phone\":\"tel\"})"
   ],
   "id": "cf02cb6e10cdaff9",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "##通过索引获取数据",
   "id": "a67eaf325e3af1ee"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "new_dataframe=pd.DataFrame(np.random.randn(5,4),columns = list('ABCD'),index=list('abcde'))#random.randn是NumPy库中的一个函数，用于生成服从标准正态分布的随机数。这里生成了一个5行4列的二维数组。\n",
    "print(new_dataframe)\n",
    "#直接通过索引取的是某列数据\n",
    "print(new_dataframe[\"A\"])#直接通过索引获的是某一列的数据,返回Series对象，原来的行索引成为它的索引\n",
    "print(new_dataframe[['A']])#返回一个DataFrame对象只有这一列"
   ],
   "id": "df5ee9c9c7941043",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "#取行数据时要用loc方法（通过索引标签）或iloc方法（通过索引位置）\n",
    "print(new_dataframe.loc[\"a\"])#使用loc方法获取某一行的数据，返回Series对象，原来的列索引成为它的索引\n",
    "print(new_dataframe.iloc[0])#使用iloc方法获取某一行的数据，返回Series对象，原来的列索引成为它的索引\n",
    "#切片时在loc或iloc方法中增添上列索引\n",
    "print(new_dataframe.iloc[0:3, 1:4])#索引切片\n",
    "print(new_dataframe.loc['a':'c', 'B':'D'])#索引切片"
   ],
   "id": "c5d6b26c0fb0a0c",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "##函数应用",
   "id": "59bffe93f65ad7c4"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "#apply方法可指定函数对哪个轴做操作\n",
    "print(new_dataframe)\n",
    "print(new_dataframe.apply(lambda x : x.max()))#apply默认函数对axis=0做操作（行索引消失）即作用在每一列上\n",
    "print(new_dataframe.apply(lambda x : x.max(), axis=1))#apply指定函数对axis=1做操作（列索引消失）即作用于每一行上"
   ],
   "id": "f571afd3b604f80d",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "#map方法使函数作用于每一个元素\n",
    "print(new_dataframe.map(lambda x : '%.2f' % x))\n",
    "print(np.abs(new_dataframe))#绝对值函数"
   ],
   "id": "3d5f1063194d23f",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "##处理缺失数据和异常数据",
   "id": "2b29af6c708a861c"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": "print(my_dataframe)",
   "id": "8abc41e473d2727f",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "#isnull来判断是否有空的数据\n",
    "print(my_dataframe.isnull())"
   ],
   "id": "81d6c764ea2b222a",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "#dropna来删除缺失数据\n",
    "print(my_dataframe.dropna(subset=['age']))#subsets参数指定列索引，表示这列的某个数据为空时，就删除这个数据所在的行。\n",
    "print(my_dataframe.dropna(axis=1))#某列(axis=1)有空数据就删除该列，axis=0表示行(默认)"
   ],
   "id": "5b7ed3287c3491f1",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "#fillna来填充缺失数据（一般用均值，中位数，众数填充）\n",
    "print(my_dataframe.fillna(55.))#所有空数据用55填充\n",
    "print(my_dataframe.iloc[:,2].fillna(55.))#先把2列取出来单独对它的空数据用55填充。之后可以再赋值回原来的列实现单独对某列的空数据填充。\n",
    "#replace方法可以用其他值替换缺失值或异常数据\n",
    "print(my_dataframe.replace(np.nan, 88))\n",
    "print(my_dataframe.replace(32, 99))#把32替换为99,返回新的DataFrame对象。若要在原DataFrame对象上修改，则需要加上inplace=True。\n"
   ],
   "id": "18439020216c54c0",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "##统计计算",
   "id": "fb0cf0b8b47d22a1"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "print(new_dataframe)\n",
    "#常用 sum, mean, max, min。其中 axis=0按列统计(默认)，axis=1按行统计，skipna表示跳过缺失值(默认为True)\n",
    "print(new_dataframe.sum())\n",
    "print(new_dataframe.max(axis=1))\n",
    "#describe方法可以产生多个统计数据\n",
    "print(new_dataframe.describe())"
   ],
   "id": "32d6adb08658a8af",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "##分组(groupby)与聚合(如sum,mean,max等)结合使用",
   "id": "e40372c2279d87b6"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "my_dict={'key1' : ['a', 'b', 'a', 'b','a', 'b', 'a', 'a'],'key2' : ['one', 'one', 'two', 'three','two', 'two', 'one', 'three'],'data1': np.random.randint(1, 10, 8),'data2': np.random.randint(1, 10, 8)}\n",
    "df1=pd.DataFrame(my_dict)\n",
    "print(df1)\n",
    "result=df1.groupby(\"key1\").max()#按key1列分组后求最大值\n",
    "print(result)\n",
    "result=df1.groupby(\"key1\").max().add_prefix('max_')#分组聚合后可以增加前缀以区分\n",
    "print(result)\n",
    "result=df1.groupby(\"key1\").transform(\"max\")#使用transform分组聚合后返回的DataFrame的形式是聚合前的\n",
    "print(result)"
   ],
   "id": "985e305e15b7866f",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "##数据连接（pd.merge）:根据两个DataFrame的共同列或行索引把其行连接起来",
   "id": "6291883657cd4ab6"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "df1 = pd.DataFrame({'key': ['b', 'b', 'a', 'c', 'a', 'a', 'b'],'data1' : np.random.randint(0,10,7)})\n",
    "df2 = pd.DataFrame({'key': ['a', 'b', 'd'],'data2' : np.random.randint(0,10,3)})\n",
    "print(df1)\n",
    "print(df2)\n",
    "print(pd.merge(df1,df2))#默认时会使用相同的列名连接，连接方式是内连接即结果中的键是交集(一个df有另一个df没有的键都不显示)\n",
    "print(pd.merge(df1, df2, on='key'))#参数on指定连接的列名\n",
    "print(pd.merge(df1,df2,left_on='key',right_on='key'))#left_on指定左df1连接的列名，right_onz指定右df2连接的列名。\n",
    "print(pd.merge(df1, df2,left_index=True,right_index=True))#左df1和右df2都拿行索引来连接\n",
    "print(pd.merge(df1,df2,how=\"outer\"))#how参数指定连接方式，outer表示外连接即结果中的键是两个df的并集。\n",
    "print(pd.merge(df1,df2,how=\"left\"))\n",
    "#另外”left“表示左连接即结果表中包含左表的所有行，以及右表中与左表匹配的行，对于右表中没有匹配的行，右表列的值将为NaN。right“表示右连接。”inner“表示内连接（默认）。"
   ],
   "id": "d0986dd8bab57547",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "#merge后对于相同的列名会自动加上_x和_y后缀以区分，也可以通过参数suffixes来指定后缀。\n",
    "print(pd.merge(df1,df2,left_index=True,right_index=True,suffixes=('_left','_right')))"
   ],
   "id": "ec60586ccbb013cb",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "##重复数据处理",
   "id": "7c2bd83b5fd0061c"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "#duplicated方法判断是否有重复数据\n",
    "df1=pd.DataFrame({'data1':['a']*4+['b']*4,'data2':np.random.randint(0,4,8)})\n",
    "print(df1)\n",
    "print(df1.duplicated())#返回布尔型Series表示这行是否为重复行"
   ],
   "id": "1a3922da850baa4a",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "code",
   "source": [
    "#drop_duplicates方法删除重复数据。duplicated认为空值和空值相等。\n",
    "print(df1.drop_duplicates())#默认保留第一次出现的重复数据，默认根据全部列的情况来删除重复数据，可以设置keep='last'保留最后一次出现的重复数据。\n",
    "print(df1.drop_duplicates('data2'))#可指定按某些列判断是否重复并删除重复数据。"
   ],
   "id": "3f8297e5b675413c",
   "outputs": [],
   "execution_count": null
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
